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ABSTRACT 



This study considered how 25 highly diverse after-school 
programs with funding of $5.6 million were evaluated during a 10-month 
period. The paper describes the evaluation methodologies used and determined 
which methodologies were most effective within a diverse and political 
context. The Bayview Fund for Youth Development (name assumed for purposes of 
this paper) was the result of a ballot initiative to set aside a large sum to 
fund programs for children and youth. The Fund supported 25 programs, 4 for 
children aged up to 5 , and 21 for children and youth aged 6 to 20. Overall 
the 25 programs implemented a total of 369 different program activities. 
Because of the diversity of program activities and the requirements of city 
administrators, the evaluation approach used methodologies from the 
empowerment evaluation model of D. Fetterman (1996) and traditional 
evaluation strategies. Both fund accountability and individual program- level 
accountability were studied. Traditional evaluation methods included project 
director interviews, program observations, parent focus groups, and exit 
interviews with project directors at the close of the evaluation. Empowerment 
evaluation methods included interviews with participating youth and feedback 
meetings with grantees and fund and city administrators. The most effective 
methods were the use of youth evaluators, the clustering of similar types of 
groups, and the use of traditional methods for collecting data in a 
systematic way. An essential first step is to identify the key stakeholders, 
assess their levels of accountability, and understand their respective 
interests in the evaluation process. Using youth evaluators and conducting 
two-way feedback meetings were important in building trust in the divers and 
political environment of these initiatives. A major finding from the 
evaluation was that a systematic and standardized information management 
system should be established for all programs. (Contains 12 references.) 
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Introduction 



Accountability is traditionally believed to be the sine qua non of good program 
evaluation. In an assessment of the 21 st Century Community Learning Center Programs, 
the U.S. Department of Education (1998) stated "it is essential that comprehensive 
evaluation take place to assess the quality of these programs and their ability to meet the 
promises they make." Yet, without trust and a common understanding of evaluation and 
its benefits for program design and implementation, evaluators may be viewed as the 
enemy of community based organizations, thereby impeding the evaluation processes and 
limiting evaluation results. This is particularly true of new after school programs in 
urban areas created by, or in response to, funds made available from philanthropic 
organizations, and federal, state and local funding. 

In this paper we discuss how 25 highly diverse after-school programs with funding of 
$5.6 million were evaluated during a 10-month period, from December 1999 until 
September 2000. It describes what types of evaluation methodologies we used and which 
ones were most effective within a diverse and political context. 



The five member evaluation team conducted a city-wide process evaluation of all 
programs funded during Year 2 of the Bayview Fund for Youth Development (BFYD). 
For purposes of confidentiality, we will not use the actual name of the Fund or the city in 
which the Fund was established. While program implementation began in July 1999, due 
to administrative and contractual delays, we were not able to start the evaluation process 
and data collection until December 1999, six months after program implementation. The 
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majority of BFYD after-school programs were community-based, and managed by 
individuals from minority groups — the primary target group of the Bayview Fund for 
Youth Development. Given the amount of funding made available from the City, and the 
lack of financial resources generally attributed to after-school programs for minority 
groups in this city, competition for limited resources was substantial, which also affected 
the evaluation processes and outcomes. 

EL Background and Overview of the Bayview Fund for Youth Development 

The Bayview Fund for Youth Development was the result of a successful ballot 
initiative to set aside almost 3 percent of the annual budget of a large metropolitan, urban 
area in northern California to fund after-school programs for children and youth ages 0- 
20. The primary purpose of the initiative was to make the City a better place for children 
and youth. The life span of this Fund was 10 years, and an important aspect of this 
legislation was the requirement that young people play a meaningful role in the oversight 
and implementation of the Fund. The initiative identified a variety of short-term and 
long-term “expected results,” which were detailed in the initiative’s Strategic Plan as 
follows: 

1) Connection to caring adults 

2) Safe places 

3) Literacy 

4) Cultural and physical activities 

5) Child care and child development 

6) Young children ready to learn 




3 



4 



7) Job training and work 

8) Community engagement 

9) Successful transition from juvenile justice 

The Request for Proposal (RFP) for Year 2 indicated the following three “expected 
results,” which were also the key outcome variables for the 25 BFYD programs that we 
evaluated: 

1) Safe places; 

2) Connection to caring adults; and 

3) Community engagement 

The Initiative was overseen by a 19-member Citizens' Committee for the Fund (CCF), of 
which nine members were youth under the age of 21. The CCF was responsible for 
setting policy for the Fund, developing a strategic plan every four years, preparing 
requests for proposals (RFP), and awarding BFYD funds to organizations submitting 
successful proposals. 



HI. Program Characteristics of BFYD 

Of the 25 BFYD funded programs, 4 served children ages 0-5, and 21 served 
children and youth ages 6-20. Overall, the 25 programs implemented a total of 369 
different program activities. Of these, roughly 16% were in the arts, 14% in youth 
leadership development, and 11% in community service. The remaining programs were 
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fairly equally distributed between employment training, sports, mentoring, life skills 
development, and tutoring. 

During the one-year program cycle we evaluated, we estimated that 6,000 children and 
youth participated in BFYD programs. BFYD programs had no eligibility requirements 
for participation. While some programs were structured on an informal drop-in basis, 
others used a 14-week curriculum-based framework. The broad variety of programs was 
also reflected in the number of children and youth each program served, ranging from 
120 to 1,200 participants, and the diversity of target populations, which included Native 
American children and youth, lesbian, bisexual and gay youth, deaf and hearing impaired 
children and youth, and foster children, among others. 

In addition to the various program activities and target audiences of the BFYD programs, 
we also included the three major program designs of the Fund into our evaluation 
process: 

1) Individual organizations, partnerships and collaboratives implemented a variety 
of after-school programs, and the BFYD awarded roughly 3.1 million dollars to 19 
such programs. 

2) Community Clusters were “clusters” of agencies that were either neighborhood- 
based or “community of interest-based.” Community Clusters involved individuals, 
service providers and community institutions, and the BFYD awarded roughly $1.5 
million dollars to three such programs. 



3) Youth-Initiated Projects focused on youth-to-youth grantmaking and youth 

leadership development. Twenty percent (nearly $1 million) of the Fund was awarded 
to three youth-initiated projects. 

IV. Theoretical Framework of Evaluation 

When conducting a city-wide evaluation of after-school programs, various 

contextual factors must be considered in the evaluation design, as they affect what 
methodologies are most effective. Guba and Lincoln (1989) correctly noted that 
identifying the different stakeholders involved and assessing their different levels of 
accountability and interest in the evaluation process is essential, including stakeholders at 
risk of not being refunded, and those who would benefit from evaluation results. Fear of 
negative evaluation findings and a lack of trust among non-profit agencies competing for 
limited resources can affect program administrators’ willingness to cooperate with 
evaluators. In addition, Flaxman and Orr (1996) asserted that most youth programs are 
small, local, service-driven, and improvised -- conditions that further affect effective 
evaluation methodologies. These contextual factors all affected our evaluation design 
and methodologies for the Bayview Fund for Youth Development. 

Traditionally, scientifically rigorous evaluation calls for analytic objectivity and employs 
methods, such as: identifying process and outcome variables, conducting pre- and post 
tests and comparing control and intervention groups. For the evaluation of the Bayview 
Fund for Youth Development, the use of such tools was not feasible due to the following 
reasons: 
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. First, the evaluation began six months after the programs had begun. As a result, we 
were not able to collect baseline data against which to compare post-test data to 
measure impact or change. 

. Second, the 25 programs were highly diverse and complex with respect to program 
design, activities and objectives. Therefore, while the Fund and the RFP identified the 
three overall “expected results,” outlined above, individual programs identified a 
variety of short-term objectives for their many program activities. Ideally, we would 
have measured whether and to what extent each of the 369 implemented program 
activities met or were progressing toward each individual program objective. Due to 
time constraints and limited financial resources for the evaluation (and the lack of 
baseline data), we implemented an evaluation process that sought to measure the 
quality of program implementation, rather than impact. 

• Third, it was essential to include the various stakeholders in the evaluation process. 
We evaluated a city fund, which was established and largely controlled by concerned 
citizens and program managers who competed for limited resources for their after- 
school programs. Specifically, the majority of program staff represented their target 
populations, drawn primarily from ethnic minority groups, low-income communities, 
and other minority groups such as gay, bisexual, deaf and hard of hearing, amongst 
others. As these groups have been historically marginalized and oppressed by 
dominant groups, such as government officials and others in positions of authority, 
ignoring their cultural milieux by relying solely on traditional evaluation methods 
would have been less effective in assessing to what extent the programs were 
working. 
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To tailor our evaluation design to BFYD’s diverse community-based organizations and 
the requirements of city administrators, we included methodologies from Fetterman’s 
empowerment evaluation model (1996) and traditional evaluation strategies, such as 
implementation of a Management Information System (MIS) and a quantitative outcomes 
survey for youth and parents (Rossi and Freeman, 1993). 

Empowerment Evaluation and Traditional Evaluation Methods 

The essence of empowerment evaluation is to shift the locus of control from 
evaluators and city administrators to program managers and community-based 
organizations in order to increase their capacity (Diaz, 2000; Fetterman, 1996). 

Fetterman (1996) pointed out that in empowerment evaluation, the evaluator becomes a 
facilitator who coaches program administrators to improve their programs, build capacity 
and create program sustainability. With respect to traditional evaluation strategies, Rossi 
and Freeman (1993) have shown that evaluation must be based on a systematic approach 
to data collection and analysis, including maintaining a management information system 
(MIS), and monitoring the delivery of services to target populations. Such traditional 
evaluation methodologies and data collection strategies provide essential information for 
future evaluation of the longitudinal effects of participation in after-school programs, on 
youth academic achievement, the creation of safe learning environment and community 
engagement. Below we describe why and how we integrated empowerment evaluation 
methods and traditional evaluation strategies in evaluating BFYD’s diverse after-school 
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programs. 



V. Objectives of BFYD Evaluation 

Given the context of the Bayview Fund for Youth Development, the fact that we 
were not able to collect baseline data, that programs were so diverse that we could not 
identify a common set of process indicators and outcome variables to measure short-term 
results and program objectives, and the intrinsic political context of the program, 
including program participants and staff, we identified the following objectives for the 
evaluation: 

1 . To document the quality (rather than impact) of program implementation, 

2. To assess the extent to which agencies were progressing toward the Fund’ s three 
major expected results; 

3. To develop the capacity of grantee agencies by involving them in the evaluation 
process, and use evaluation information to improve the quality of the programs they 
deliver; 

4. To inform policy makers about the status of the Fund’s programs, especially 
individuals who influence funding decisions for the year; and 

5. To assist Fund administrators in monitoring the progress of BF YD-funded programs 
and assessing program needs. 

In this paper, we will discuss the first three objectives of the evaluation, for which we 
used the following methodologies. The fourth and fifth objectives are discussed in "After 
School Programs, Implications for Urban Education" by Walker-Moffat (2001). 
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VI. Methods 



Levels of Evaluation 

We distinguished two levels of accountability for our evaluation of 25 BFYD- 
funded programs: 1) Fund accountability, and 2) individual program-level accountability. 
We used a variety of qualitative and quantitative evaluation methodologies for both 
individual program-level and Fund-level evaluations. In addition to these two levels of 
accountability, nine youth evaluators evaluated 16 programs using empowerment 
evaluation as their methodological framework. 

Fund Evaluation 

We collected and analyzed quantitative and qualitative data for all 25 programs 
from December 1999 until September 2000. To evaluate whether the Fund was 
“working” and whether funded agencies were collectively progressing toward the Fund’s 
overall expected results, we developed two quantitative survey instruments. 

• The Youth Survey instrument attempted to measure the extent to which programs 
provided services that met the three overall expected results of the Fund: 1) to create 
safe places, 2) to have participants connected to caring adults, and 3) to engage youth 
in the community. The instrument contained 14 items, and was pilot tested prior to 
distributing the survey to 1,450 participants. Participants were randomly selected 
from a Participant Reporting Form that each agency was required to complete by a 
specific date. Based on the information available at the time of survey administration, 
the unduplicated total number of youth ages 6 and older participating in BFYD 
programs in Year 2 was about 3,000 (not including youth who participated in the 
programs offered by three grantees who did not submit participant lists). 

O 10 
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. For the youth survey, we targeted 1,500 youth, which was 50% of the population,. 

The anticipated response rate was 60%, resulting in a sample of 900. We choose 900 
as a mark, because 811 survey responses were needed to obtain survey findings at 
95% confidence interval. Of the 1,450 surveys administered, 708 completed surveys 
were returned (response rate: 48.8%). Of these, 101 youth surveys were excluded 
because the respondents were younger than 10 years of age (the age threshold we 
determined for ability to understand and fully complete the survey). The findings of 
the survey, discussed below, are based on the responses of 607 youth over the age of 
10 . 

. The Parent Survey instrument contained 19 items and included questions to assess 
program safety, connection to caring adults, and child care and child development. 

We surveyed all five BFYD programs that served children ages 0-5. Our sampling 
technique consisted of a combination between random sampling and sampling the 
entire population of parents whose children attended BFYD programs. We chose this 
combination because of the high variance in the number of children served among the 
five agencies. 

. Based on the information from program documents, the estimated population of 
children ages 0-5 was 3,358 (N). Our goal was to sample 5% (n=168). Due to the 
difficulty in reaching parents, and of maintaining a consistent number of parents 
sampled, we increased the number of surveys to be distributed by roughly 50%. 
Accordingly, a total of 295 parents were selected for sampling, and a total of 79 
completed parent surveys were obtained by July 2000 (response rate, 26.7%). Of 
these, 17 responses were excluded because survey participants were parents of 
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children ages 10 and older. Our findings were based on a total of 62 survey 
respondents. 

. Cluster and Youth-initiated Projects Program Designs were evaluated both on a 
Fund-level, meaning collectively, and individually, to assess the overall effectiveness 
of these program designs. We collected quantitative and qualitative program data 
from the following sources to assess to what extent the Cluster and Youth-Initiated 
Project models were working: 

. progress reports; 

. self-evaluation reports; 

. program observation; 

• youth evaluator data; and 

. data from project director introductory and exit interviews. 

Finally, we also designed and implemented two standardized reporting forms to assist 
BFYD grantees with their management information systems (MIS). For our evaluation, 
the objective of the Participant Reporting Form was to gather information on how many 
children and youth participated in BFYD-funded programs during Year 2. Reporting data 
also included demographic information on ethnicity and age of program participants. This 
data was also used to assess whether the youth and parent survey respondents constituted 
representative samples of the Fund’s participants. The Service Activity Form generated 
data on what type of program activities were implemented during BFYD’s Year 2 
funding cycle. Both forms were distributed during the third and fourth quarter of the Year 
2 funding period and therefore did not contain data for the entire funding period. 
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Individual Program-Level Evaluation 

We used a variety of qualitative and quantitative data sources to assess the quality 
of individual grantee performance (evaluation objective 1) and to develop the capacity of 
grantee agencies (evaluation objective 3). As discussed above, we combined traditional 
evaluation methods with empowerment evaluation strategies to increase grantee capacity 
and involve them in the evaluative process. 

Traditional Evaluation Methods 

Project Director Interviews. In December 1999 and January 2000 we conducted 
site visits and introductory project director interviews with all 25 lead agencies. For the 
introductory interview we asked three key questions: 1) What did you say your program 
was going to implement in your scope of work? 2) How did you say you were going to 
evaluate your activities and outcomes, and 3) How do you perceive the importance of 
keeping data on individual participants for you own records, and for this evaluation 
project? This information was critical for assessing the overall program objectives, and 
determining what methods BFYD grantees used to collect attendance data of program 
participants. 

Program Observations. Between February and May 2000, the evaluation team and the 
youth evaluators conducted three observations of individual program activities of all 
agencies in their second year of funding, Clusters and Youth-Initiated-Projects. The 
observation data provided important information about how well the activities were 
implemented. We analyzed observation data by reviewing the analytical notes and by 
conducting multiple readings of field notes to assess common patterns and themes. 
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Parent Focus Groups. In March 2000, we conducted parent focus groups with each 
agency in its first year of funding that served children ages 0-5. Youth evaluators were 
present during the parent focus groups in order to introduce them both to the grantees and 
to this method of gathering qualitative data. Focus groups were tape recorded with the 
permission of the participants, and analyzed for patterns and themes. The data provided 
critical information on whether the participants were satisfied with the services they 
received, and how they viewed the quality and impact of these programs. 

Project Director Exit Interview. In June and July 2000, we conducted exit interviews 
with project directors of all 25 BFYD-funded agencies. The main purpose of these 
interviews was to encourage the project directors to be self-reflective about the successes 
and challenges they faced during the past year’s program implementation. The interviews 
focused on three main topics: 1) program objectives, 2) self-evaluation and data 
management, and 3) fiscal accountability. 

Empowerment Evaluation Methods 

We implemented two key empowerment evaluation strategies with the aim of 
developing the capacity of grantee agencies by involving them in the evaluation process. 
Youth Evaluators. Nine youth evaluators conducted youth-to-youth interviews and 
participant observations of a total of 16 BFYD programs. The youth evaluators were 
themselves program participants of BFYD programs and were between the ages of 15 
and 18. Between March and June 2000, the youth evaluators conducted youth-to-youth 
interviews with all first year grantees, except agencies serving children ages 0-5. The 
interview instrument contained 15 open-ended questions that ranged from how the 
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participants became involved in the program, to where the participants would be if the 
program did not exist. In addition, the youth evaluators also conducted participant- 
observations of program activities with the Cluster agencies, Youth-Initiated Projects, 
and all second year grantees. 

Feedback Meetings with BFYD Grantees, CCF Members and City Administrators. 

In May 2000, mid-way through the evaluation, we conducted a series of feedback 
meetings with grantees, CCF members and City administrators to discuss interim findings 
of the evaluation. We held different meetings based on individual program design, target 
populations served and program activities. The overall objective was to inform the 
grantees of our preliminary evaluation findings based on data gathered from the 
individual program-level evaluation process. We also provided technical assistance with 
respect to implementing a systematic and standardized management information system 
(MIS) and self-evaluation processes. 

VII. Findings of City-wide Evaluation 

Results from Fund-level Evaluation 

The overall objective of the Fund-level evaluation was to assess the extent to 
which agencies were progressing toward the Fund’s three major expected results. 

Overall, on the Fund-level, findings from the youth and parent outcome surveys 
suggested that the majority of Bayview Fund for Youth Development programs provided 
safe places for youth participants, connections to caring adults, and increased youth 
participants’ community engagement. In addition, the majority of programs that served 
children ages 0-5 improved parents’ knowledge and confidence regarding child care and 
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child development. Youth survey data was based on the responses of 607 youth over the 
age often, and parent survey data was based on the responses of 62 parents. Youth 
evaluation data is based on youth-to-youth interviews with six (6) BFYD programs, and 
youth participant-observation of ten (10) BFYD programs. 

Safety 

For the youth and parent surveys, safety was defined in terms of physical safety, 

psychological safety and safety of participants’ belongings. Overall, the youth in all 

ethnic groups felt that they were safer at their program site than in their own 

neighborhood or at school. 82% of the youth respondents claimed that the programs were 

the safest place for them to be with respect to fear of “getting beaten up.” This 

quantitative data was also supported by youth evaluators’ participant-observation and 

youth-to-youth interview data. For example, one youth evaluator described the sense of 

safety of a grant writing workshop in which he participated as follows: 

The kids were very safe in the classroom where they held their meetings. Other 
staff members at the school were looking out for them when they took their break. 
These other staff members were not a part of the program, but they still watched 
out for the kids. 

Similarly, another youth evaluator, who participated in an arts and crafts workshop of a 

community-based organization, observed the following: 

The participants definitely felt safe. They looked very comfortable in their 
surroundings. They didn’t look like they had a care in the world while they were 
there. 

Parents also believed that the program sites were safe. Overall, findings from the Child 
Safety Subscale 1 indicated that over half of the parents (59%) reported that the program 



1 Description data from the parent survey data were generated for all variables; no comparative analyses 
were conducted because this survey was only administered as a post-program survey. Subscales were 



site was very safe for both the child’s health and for his or her belongings. In addition, 
87% of the respondents reported that someone was available “all the time” or “most of 
the time” in the program to help his or her child when he or she needed help. 

Connection to Caring Adults 

For the youth outcome survey, we defined connection to caring adults as having 
an adult to talk with about personal problems and having an adult around who really 
cared about youth. Of the 607 youth who completed the outcome survey, 90% came from 
homes with adults who cared about them. While 72% of the respondents also found a 
connection to caring adults at the program site, 7% reported that they did not have any 
adults anywhere to talk with about their personal problems. 

In general, the qualitative data gathered by the youth evaluators on this outcome 
variable also indicate that the majority of youth participants have a good connection to an 
adult staff member at the program site. For example, one youth evaluator noted the 
following impression after having participated in a business management workshop at a 
local high school: 

Very close connection. The staff made sure to give the youth ample time to 
answer the question and was [sic] very supportive. It almost seemed like a 
mother/son relationship. 

Another youth evaluator conducted a participant-observation of an after-school activity 
that involved the construction of scientific objects. He described his observation as 
follows: 

The connection with the staff was great. They acted more like friends that you 
would meet in life rather than your supervisors (bosses). The staff treats the youth 



created by combining key outcome variables to create Child Safety Subscale and the Parent Involvement 
Subscale. 




17 



13 



very well. They take in the youth’s ideas on projects, and help them with it. The 
staff has fun while helping the youth. The youth get a lot of encouragement from 
the staff. The staff often tells the youth that their project is good or interesting. 
The youth trust the adult/staff around them because they seem happy and excited 
about the projects when around these people . . . 

Findings from the parent survey are similar to the youth survey findings with respect to a 
child’s connection to caring adults. Nearly all of the parents thought their children (ages 
0-5) developed a connection to a caring adult. 98% of all respondents reported that their 
children seemed comfortable at the program site. In addition, 95% reported that their 
children seemed comfortable with the other children at the program site. While this data 
did not reflect a definite connection to caring adults, it nevertheless highlighted the fact 
that the program site was a place where children made positive connections. 

Community Engagement 

As with the other two major outcome variables, or “expected results” of the Fund, 
the findings for community engagement from the youth survey were positive. 75% said 
that they either “agree” or “strongly agree” that they are “more interested in knowing 
things” about their community as a result of their participation in the BFYD. 67% said 
they are “doing more things to make their community a better place to live.” Only 8% 
said they did not participate in any community oriented activities. 

The youth evaluators also assessed the level of community engagement among 
BFYD participants in their youth-to-youth interviews. For example, of the six interviews 
conducted, each with 3 participants, 10 youth participants described various community 
activities in which they participated as a result of their participation in the program. For 
example, three youth attending an after-school tutoring program implemented by a 
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Vietnamese community-based organization responded as follows to the youth evaluator’s 
question regarding community involvement: “I give back to the community by 
volunteering my time.” Another said, “I bring other students into the program, so they 
can get help with their homework.” The third of the three participants stated: “we are 
helping kids, we help poor people, and we go to church.” 

Child Care and Child Development 

The majority of parents said they learned about child care and child development 
as a result of their children being in a BFYD-funded program. 80% of the respondents 
felt more confident about being a parent or a guardian. 

Regarding parent involvement in program planning, implementation and 
evaluation, findings were not as positive. For example, while 67% reported that their 
child’s program offered parent workshops, nearly one third (30%) reported that they did 
not know whether such workshops were offered. 

Results from Project-level Evaluation 

One key objective of the project-level evaluation was to document the quality 
(rather than impact) of program implementation and grantee performance. As discussed 
above, we conducted observations of program activities, focus groups with parents of 
children attending BFYD-funded programs, and in-depth document reviews to assess 
agency performance. In general, from a service delivery perspective, the findings were 
positive. The majority of the agencies served the target population, and program activities 
were well implemented. However, a program area that needs improvement concerns 
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program monitoring and administration, including record keeping of individual 
unduplicated program participants, budgets/expenditures, and self-evaluation. 

V III Methodologies: Which are effective in a diverse and political environment? 

Analytic Context 

The BFYD programs were far from homogenous. Few of the programs had 
clearly identified outcome measures or indicators. There was no baseline data, and the 
timeframe of the evaluation was short (10 months), given the diversity and overall 
number of the 25 agencies’ program activities (over 360 program activities were 
implemented during Year 2). 

To answer the question of which methodologies were most effective within the 
given context, it is important to understand the various key stakeholders involved and 
their objectives and interests in the evaluation results. The four key stakeholders in this 
evaluation were: the BFYD grantees, the city administrators, the Citizens’ Committee for 
the Fund (CCF), and the evaluators. Each had a different agenda regarding the 
evaluation, and at times opposing interests in the outcome of the evaluation. For the 
BFYD grantees, the primary interest at stake was refunding. The BFYD funding cycle 
was one only year, resulting in the fact that current and potential grantees have to submit 
grant proposals on an annual basis. Proposals are evaluated on an annual basis, and 
current grantees did not receive preferential treatment for refunding based on their current 
grantee status. 2 In addition, the CCF and city administrators did not inform the grantees 
prior to the evaluation process as to how the results of the evaluation would be used. 



2 Of tlie 87 proposals that were submitted for Year 2 funding, 25 were selected. Of these 25, 14 were first 
year grantees, and 1 1 were 2 nd year grantees. 



Accordingly, grantees had no clear incentives for collaborating with the evaluators. In 
fact, some grantees expressed open hostility toward the evaluators, who to them 
(members of minority groups, which historically had been struggling to gain a voice in 
Bayview) represented yet another dominant authority group that jeopardized their 
funding source (even though the evaluation team was diverse). 

According to the Fund legislation, the CCF was required to submit an “annual 
independent process outcome evaluation” every year to the City Council. CCF and City 
Council members had two major objectives regarding the evaluation: first, to be informed 
regarding refunding decisions and monitoring strategies of the overall Fund, second to be 
informed about policy issues affecting the Fund. Consequently, both the CCF and the 
City Council were highly interested in seeing “hard data” on the progress of Year 2 
grantees, including the implementation of a standardized MIS. 

The evaluation team consisted of two sub-teams: the five adult evaluators, who 
conducted the Fund-level and individual project-level evaluations, and the nine youth 
evaluators, who were also program participants of BFYD programs. The youth 
evaluators’ overall goal was to learn more about the programs’ implementation processes 
and their progress toward the three major expected results outlined in the Strategic Plan, 
from a youth perspective. As outlined above, our objectives for the evaluation were to 
assess the quality of grantee performance, to assess the progress toward the major 
strategic objectives, and improve grantee capacity. 
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Methodologies for a Diverse Environment 

The 25 programs were highly diverse. A parent or youth seeking a particular type 
of after-school activity should have been able to find it in Bayview. The diversity of the 
program designs (Cluster design, Youth-initiated programs, individual agencies, 
partnerships and collaborative), the diversity of the target population, and short-term 
objectives of individual programs (or lack and/or inconsistency of program activities with 
short-term objectives), affected the rigor of the both the parent and the youth survey 
design. Within such a diverse context, the “lowest common denominator” of all BFYD 
programs were the three “expected results,” which are not readily quantifiable and 
measurable, especially without baseline data. 

Based on the city-wide evaluation of the BFYD, we provided the following 
recommendations: 

Youth Evaluation For a diverse programmatic environment, evaluation 
methodologies should include a strong empowerment evaluation component. Youth 
evaluators should be included in any evaluation process of urban after-school programs 
as they can collect a rich set of process evaluation data of program implementation not 
accessible to adult evaluators. Youths should be selected from program participants and 
represent the youth being served. This increases trust and shifts the power of the outside 
evaluator to program participants. Youth should be trained in the various evaluation 
methodologies, which demystifies the evaluation process and creates the opportunity for 
capacity building (Fetterman, 1996). Even though the youth evaluators were not program 
staff involved in program development and implementation, the participatory process of 
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training youth program participants to evaluate their program created trust among the 
culturally and ethnically diverse program staff of the BFYD funded programs. 

Self-Evaluation. Program staff should be trained in evaluation methods so that 
individual programs can effectively evaluate their own performance. While BFYD 
grantees had identified their own program objectives and indicators, it was not 
uncommon that the indicators did not match program activities, which in turn did not 
necessarily lead to the identified objectives. It is essential that program staff be trained to 
conduct self-evaluation, including evaluation methods to measure outcome In this way, 
the diversity of program activities and participants can be included in each agency’s 
individual program self-evaluation design. Such a procedure requires time, human and 
financial resources. Program staff not only need to be trained by evaluators, but also 
should collect data at key points throughout the life of a program (Flaxman and Orr, 
1996). 

Cluster Evaluation. Within the context of a diverse environment, with a variety 
of programs designs, activities and objectives, programs should be evaluated on a cluster 
basis - programs with similar objectives and program activities can be evaluated 
collectively and comparatively (Kellogg Foundation, 1998). The overall goal of the 
cluster evaluation method is to gather data on programs with similar objectives to identify 
common threads and themes, that having cross-confirmation, take on greater significance. 
Our evaluation of the three youth-initiated and community cluster programs funded by 
the BFC indicates that such a comparative evaluation allows for a more in-depth analysis 
and assessment of program implementation. 
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Methodologies for a Political Environment 

Competition for the Fund’s limited resources was substantial, which affected the 
level of collaboration between the grantees and evaluators, and the overall evaluation 
process, including data collection. In addition. City administrators and CCF members 
provided opposing incentives to the BFYD grantees and the evaluators: grantees had to 
meet the service requirements of the Fund, which set only 5% aside for program 
administration, including evaluation, and the evaluators had to assess Fund effectiveness. 

Historically, there have been a variety of barriers to collaborative relationships 
between research evaluators and community service providers (Gomez and Goldstein, 
1996). Money, Ownership, Rigor and Time - the MORT Syndrom - have been identified 
as some of the key differences between research evaluators and community service 
providers that contribute to historical mistrust (Gomez and Goldstein, 1996); money, 
because community-based organizations (CBOs) are often under financial constraints for 
providing services and self-evaluations; ownership, because decisions regarding the 
evaluation design and interpretation of findings are often sources of controversy in 
community-based evaluation, such as the BFYD; rigor, because the disparity between a 
scientifically rigorous evaluation process and the need of the CBO grantee is often 
significant, and perceived as unrealistic on either side; and time, because service 
providers frequently have the immediate need to provide services to their clients, while 
evaluators tend to concentrate on macro-level program findings that will benefit a large 
number of participants sometime in the future. Within the political context of the 
Bayview Fund for Youth Development, the MORT Syndrom was undeniably present. In 
an effort to identify methodologies that address some of the perceived and real 




, 24 



25 



differences causing the MORT Syndrom and creating barriers to collaborative 
relationships, we recommend the following evaluation methods. 

Introductory and Exit Interviews with Project Directors. Structured open-ended 
introductory and exit interviews should be conducted with key program staff, such as 
project directors. The introductory one-on-one interview is an effective way to discuss 
program objectives and activities, and inform the grantee about the evaluation process. 
Introductory interviews are also effective in assessing what type of self-evaluation 
processes, if any, the agency plans to implement during the program implementation 
phase. Exit interviews provide an essential opportunity to gather feedback on the 
evaluation and program implementation process. 

Evaluation Feedback Meeting. Feedback meetings should be held at least once during 
the evaluation process to communicate interim evaluation findings to all stakeholders. 
Also, grantees with common program objectives and activities should be gathered 
together so that they have the opportunity to exchange program implementation and 
evaluation strategies, within a context where outside evaluators, city administrators and 
CCF members are present. This method develops the capacity of grantee agencies by 
involving them in the evaluation process, and provides evaluation information to improve 
the quality of the programs they deliver. 

MIS and Technical Assistance: The Participant Reporting Form was perhaps the most 
controversial method of the entire evaluation process. Some of the service providers 
resisted documenting the number of participants served (with participant ID), particularly 
participants’ ethnicity and age. For this method to be effective within a diverse and 
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political context, it is essential that technical assistance is being provided by the 
evaluators and the names of youth participants remain anonymous. 

IX. Conclusion 

In a diverse and political context the most effective methods for evaluation of 
urban after school programs include the clustering of similar type groups, the use of 
youth evaluators, and the use of traditional methods for collecting data in a systematic 
way. First, however, it is essential to identify the key stakeholders, assess their levels of 
accountability, and understand their respective interests in the evaluation process. This 
can be done more easily with tools of empowerment that shift the locus of control to the 
stakeholders, than with traditional tools of evaluation, such as a comparison between 
control and intervention groups or pre- and post- tests. 

In the evaluation of the Bayview Fund for Youth Development, the key stakeholders 
were the grantees, the City administrators, the Citizens' Committee for the Fund, and the 
evaluators. We used a combination of methods. To compensate for the lack of baseline 
data, we shifted the focus to an analysis of the quality of implementation, rather than of 
the outcomes of the programs. Youth evaluators from the programs conducted an 
independent evaluation using focus groups, participant-observation techniques, and 
interviews with fellow youth. The youth survey, the parent survey and MIS were the 
primary sources of information in this regard. The grantees were clustered according to 
the age groups they served and their types of administration. 




26 27 



The fact that the programs ranged from offering midnight basketball classes for young 
girls to HIV/ AIDS prevention workshops for gay, bisexual and questioning youth, to 
theater and youth opera performance programs, raises the question of how much rigor can 
and should there be for a Fund-level evaluation. In other words, is it feasible and 
reasonable to conduct a Fund-level evaluation and to expect to assess the overall 
effectiveness of the Fund? By combining such diverse programs, the result was that the 
outcome evaluation was reduced to the lowest common denominators. At the Fund level, 
the three salient features of the Strategic Plan and RFP were safety, connection to caring 
adults, and community engagement. 

A systematic and standardized management information system (MIS) should be 
established for all after-school programs, which requires the cooperation of the various 
stakeholders, and a shared interest in the collection of data. Program staff should 
document the number of unduplicated participants and various demographic information, 
including ethnicity, age and gender. In a political context, however, it is unlikely that 
grantees, the City administrators, or the Fund administrators will participate willingly 
unless they share an interest in the results. Tying the results of data gathering to future 
funding compromises the results for those who risk not being refunded or fear that 
negative findings would reflect poorly on their administrative abilities. Trust is the key 
variable. Utilizing youth evaluators from the grantee organizations and conducting two- 
way feedback meetings, are two effective methodologies that build trust in a diverse and 
political environment. 
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